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ABSTRACT 

To meet accountability challenges from a 
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(freshman satisfaction survey data, enrollment data, and academic 
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institution with a comprehensive set of outcome indicators and a 
model of the freshman experience. The integrated dataset is used to 
develop models of freshman retention. In order to focus more clearly 
on customer satisfaction, models of student satisfaction were 
developed to determine critical components in freshman satisfaction. 
Implications for institutional research include: (1) researchers can 
use integrated freshman da t abases to provide more comprehens i ve 
outcomes assessment measures; (2) integrated data providing a 
comprehensive picture of the freshman expeiience provides decision 
makers with more useful information for developing successful 
institutional strategies; (3) researchers can develop longitudinal 
databases and institution-specific models of the freshman experience; 
and (4) this integrated analysis uses a total quality approach to 
understanding the freshman experience, (Author/JB) 
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From Retention to Satisfaction: New Outcomes for 
Assessing the Freshman Year Experience 

Abstract 

To meet higher education's challenge of accountability from a customer-satisfaction 
perspective, one urban institution has developed an integrated approach to studying the fresh- 
man year experience in order to develop comprehensive outcome measures for assessing 
freshman success. Multiple sources of data (freshman satisfaction survey data, enrollment 
data and academic performance data), are integrated into a database which provides the 
institution with a comprehensive set of outcome indicators and a model of the freshman 
experience. This institution used the integrated dataset to develop models of freshme;i 
retention. In order to focus more clearly n n customer satisfaction, models of student 
satisfaction were developed to determine critical components in freshman satisfaction. 
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From Retention to Satisfaction: New Outcomes for 
Assessing the Freshman Year Experience 

Introduction 

To meet higher education's challenge of accountability from a customer-satisfaction 
perspective, one urban institution has developed an integrated approach to studying the fresh- 
man year experience in order to develop comprehensive outcome measures for assessing 
freshman success. Multiple sources of data (freshman satisfaction survey data, enrollment 
data and academic performance data), are integrated into a database which provides the 
institution with a comprehensive set of outcome indicators and a model of the freshman 
experience. This institution used the integrated dataset to develop models of freshmen 
retention. la order to focus more clearly on customer satisfaction, models of student 
satisfaction were developed to determine critical components in freshman satisfaction. 
Conceptual Framework 

Institutions of higher education nationwide have been challenged by their various 
constituents to demonstrate student success. Traditionally, these measures of success have 
included such indicators as freshman retention rates. However, because of the wave of total 
quality management influencing higher education (Sherr & Lozier, 1991; Coate, 199i), these 
traditional measures of success fall short. They do not address the issue of improving 
services to all students, including those who are retained. The question we should be asking 
ourselves is not how many freshmen re-enroll at the institution for a second year, but with 
what type of experience do we want to provide freshmen and are we meeting their 
expectations (Levine, 1988). We should focus on the entire freshman year experience to dem- 
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onstrate the success of higher education. 

To meet these challenges, the Office of Institutional Planning and Research at DePaul 
University, an urban, comprehensive institution, uses an integrated approach to analyze the 
freshman experience by linking multiple datasets to create a comprehensive freshman 
database. Using this integrated database, the institution can develop comprehensive outcome 
indicators to demonstrate success and can develop institution-specific models to enhance the 
freshman experience, including models of retention and satisfaction. 
Methodology 

The Office of Institutional Planning and Research (OIPR) developed an integrated 
freshman database by linking student survey data to other institutional data, including 
freshman Cooperative Institutional Research Program (CIRP) data, one-year attrition data, 
academic performance data and graduation data. 
Student Satisfaction Survey 

OIPR conducts the Student Satisfaction Survey in the Spring of each academic year. 
The survey, originally developed in 1990 by the Office of Student Affairs as a quality of 
service measure, has been administered by OIPR since 1991. The survey assesses student 
satisfaction with various facets of the university and includes key sets of questions focusing: 

* programs and services, 

* university learning and social environment, 

* university mission and values, 

* education preparation, 

* transfer intent, 
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* general satisfaction, 

* attitudes toward coursework, and 

* student demographic information. 
Survey Development and Sample Selection 

The Office involves the university community in the annual survey review process. 
We have developed our own mail survey instrument instead of using a commercially available 
survey because it provides the university with more institutional-specific information for 
quality improvement. The limitation of using a survey developed in-house, however, is that 
no national comparative data are available. We strive for an instrument that is sufficiently 
consistent to allow longitudinal tracking of data as well as flexible enough to address 
important topical concerns as they arise. 

All freshmen and samples of undergraduates, graduate students and law students 
receive this mail survey in the spring of the academic year. The freshman surveys are 
number coded to allow us to link this data with data from other sources. The initial mailing 
is followed by colorful reminder cards and additional mailings to ensure a high response rate. 
In addition, subsets of students are targeted for special reminders, such as freshmen CIRP 
respondents. A high response rate is especially important when linking multiple datasets, in 
order to insure a large pool of students who can be linked to data from other sources. For 
the 1992 Satisfaction Survey, the overall response rate was 47.3%, with 56.5% for freshmen, 
34% for undergraduates, 56.5% for graduates and 47.9% for law students. In 1993, the 
overall response rate improved to 51.8% 5 with 47.6% for freshmen, 49.6% for undergraduates, 
59.5%o for graduates and 53.6% for law students. 
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Long-Range Survey Plan 

We developed a long-range survey plan to track students who responded to the student 
satisfaction survey as freshmen (Sanders and Chan, 1993). This plan allows DePaul to 
develop a comprehensive set of attitudinal indicators spanning the breadth of the DePaul 
student experience and includes the following surveys: 

1. CIRP Survey: surveys freshmen as they enroll in the institution. This survey 
piovides information on pre-college attitudes, college achievement indicators and student 
demographic information. 

2. Student Satisfaction Survey: surveys freshmen in the spring term. This survey will 
provide information on customer satisfaction with the institution's academic and student 
support services, perceptions of the overall environment of the institution and information 
related to student retention. 

3. Rising Junior Survey. This survey, which will be completed when freshmen are in 
the fall of the junior year, will provide more in-depth information on student perceptions 
during their educational experience of their academic experiences, such as satisfaction with 
coursework, as well as student services such as academic advising and career planning. 

4. Graduating Senior Survey: surveys freshmen in the fall after graduation. This 
survey will provide a post-graduation perspective on student academic experiences, including 
academic preparation, satisfaction with the educational experience, and post-graduation plans. 

5. Alumni Survey: This survey, which will be completed two to five years after 
graduation, will provide outcome information on graduates in terms of employment or 
continued education, as well as a pot graduation assessment of the educational experiences. 
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Major Findings 

Retention Models 

An institution-specific model of freshman retention was developed drawing on the 
work of Pascarella and Terenzini (1983), by linking the 19^2 Student Satisfaction Survey data 
with the 1991-92 freshman attrition data. The overall attrition rate for these freshman was 
19.7%, up slightly from 18% in 1990-91. This gave us a sample of 540 students with both 
attitudinal and retention data; 484 students who re-enrolled and 56 who did not re-enroll. 

Non-parametric statistics indicated that there were several statistically significant 
deferences between the groups of students who re-enrolled and those who did not. Students 
who returned to DePaul were generally from in-state, had higher cumulative DePaul GPAs 
and were more satisfied with their academic experiences, level of preparation provided by 
DePaul, level of personal growth, and were more satisfied with the Admissions and 
Orientation processes. 1 

In developing a model of retention, we used logistic regression to assess which 
variables significantly increased the probability of students staying at DePaul. The analysis 
indicated that the three most important variables that improved the likelihood of freshman 
leaving DePaul were out-of-state status, low cumulative freshman grade point averages, and 
low institutional commitment (low agreement that attending this institution was the right 
decision)/ This model correctly classified only 19% of the leavers. 

The next year, we created twc, more powerful retention models, based on a linked 
dataset of the C1RP survey, the Student Satisfaction Survey, and academic and enrollment 
indicators. One model, based on the admission* indicators and the CIRP survey, focused on 
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pre-matriculation indicators of student success. The other, based on the Student Satisfaction 
survey and the students enrollment and academic information, focused on indicators from the 
collegiate experience. Although the two models provided far more information on student 
behavior and statistically they explained much more of the variance, they were still limited to 
explaining attrition and retention and did not provide policy makers guidance to improving 
services to all students (DePaul University, 1994). 
Limitations of the Retention Model 

In general, retention models provide institutions with useful information about why- 
students may stay and leave the institution. Our retention model gave us information on key 
differences between those freshmen who stayed at DePaul and those who left. However, this 
retention model was limited for several reasons. First, the merged dataset contained data for 
484 freshmen who re-enrolled for their sophomore fall and only 54 freshmen who did not re- 
enroll. This represented only 24% of the total group of students who did not return to 
DePaul. 

Second, this assessment did nothing to promote continuous improvement of student 
services and experiences nor did it focus on the needs of the consumers. The focus of this 
study was to predict students who would stay or leave DePaul, not to assess the quality of 
their experiences while at DePaul or whether DePaul met its expectations. 

Third, this model of retention did not illuminate variables in the analysis that could be 
manipulated by the institution. In addition, this study provided little information for policy 
m?.Vers in developing recommendations to improve the student experience. 

For these reasons, we shifted our focus from developing a model to predict retention to 
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developing a model of freshman satisfaction. This model allows for a more powerful analysis 
of the freshman experience by focusing on students' perceptions of their educational 
experience and allows us to develop strategies for continuous improvement and to draw more 
specific policy implications to strengthen the student experience for students who remain at 
the institution. 

Satisfaction Models 

In order to respond to new challenges in higher education by addressing the needs and 
concerns of the consumer, we chose to take this analysis a step further and examine what the 
important variables are in predicting student satisfaction. We used the 1993 Student 
Satisfaction Survey data and merged this with the one-year attrition data to provide academic 
performance information. Our sample was 484 students, 414 who were retained and 19 who 
left (with missing attrition data for 21 students). This was a freshman response rate of 47.6%. 

Using reliability analysis, we constructed 10 indices from the Student Satisfaction 
Survey (see the following page for a description of the items on each of the indices) using 
items to which over 75% of the students had responded: 

* overall academic satisfaction 

* students/social life 

* environment 

* mission 

* teaching/faculty 

* general education 

* academic support 
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Table 1 

Satisfaction Indices & Reliability 



Overall Academic Satisfaction 084) 

Academically challenged 
Education preparing for real life 
Excellent academic reputation 
Satisfied with academic experiences 
Satisfied with intellectual development 
Confident in right decision to attend DePaul 
Good academic atmosphere 

Students/Social Life (. 78) 

Close relationships with other students 

Relationships with students satisfying 

Good social atmosphere 

Social activities - something for everyone 

Clubs/organizations that match my interest 

Places to relax and meet friends 

Environment (. 74) 

Aware of other cultures 

Men and women have equal opportunities 

Minority students have opportunities 

DePaul encourages me to get involved 

I feel welcome at DePaul 

Mission (.79) 
Catholic presence 

University committed to Vincentian ideals 
Demonstrated commitment to urban mission 
Environment fosters growth 
Spirit of respect between cultures 
Open to expressing religious views 

Academic Support (.80) 
Academic advisors are available 
Meetings with advisors are helpful 
Advisors are sensitive to students' needs 

Financial Aid (.84) 
Financial aid office kept me informed 
Office answered my problems 
Tuition counselors helpful 



Support Services 084) 

Adequate copiers, of good quality 
Reasonable cost of copiers 
Enough phones on campus 

Quality of food in cafeteria good, reasonably priced 

Cafeteria is clean, pleasant 

Cafeteria staff is friendly 

Satisfied with recycling program 

Bookstore hours are convenient 

Bookstore staff is helpful 

Textbooks are available 

Library has needed materials, convenient hours 

Library personnel are helpful 

Teaching Faculty (.83) 

Quality of instruction is excellent 

Faculty instructional methods are compatible with 

my needs 

Liberal Studies Program is effective 
Easy to reach faculty during office hours 
Students/faculty take course evaluations seriously 
Students receive personal attention 
Developed close relationship with faculty member 
Outside of class interaction with faculty are positive 
Faculty are generally superior teachers 
Faculty are interested in students 

General Education Courses ( 83) 
Gen ed courses are interesting/academically 
challenging 

Gen Ed courses are offered at convenient times 
Gen Ed courses are relevant to life/area of study 
There is a good selection of course options 
Courses are generally satisfying 
I mostly enjoy taking these courses 

Admissions/Orientation (.90) 

Admissions reps/materials gave accurate picture 

Admissions was helpful 

Visit to DePaul was helpful 

Orientation made adjustment easier 

Orientation introduced me to values 

I would advise a freshman to attend orientation 

Programs were helpful 

Orientation just the right length 
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* admissions/orientation 

* support services 

* financial aid/services. 

These 10 indices and academic performance data, which included high school grade 
point average (GPA), cumulative DePaul GPAs, and composite SAT scores, were used in a 
regression analysis to determine which of these variables were important in predicting overall 
satisfaction with DePaul. Our results showed that the key variable in predicting overall 
satisfaction was overall academic satisfaction, followed by students/social life and 
environment (standardized Beta weights of .56, .18 and .11 respectively). 

We also examined the differences in overall satisfaction for several groups of students. 
There were no significant differences in overall satisfaction: 

* by gender 

* by geographic location (in Chicago/suburbs vs. out of the metropolitan area) 

* by residence hall status (currently lived in the residence halls vs. off-campus) 

* by college. 

However, there were differences in the level of overall satisfaction by ethnicity. 
Hispanic students had significantly lower student satisfaction compared to white students 
(means of 2.9 compared to 3.3). 3 

Although there were no significant differences in overall satisfaction for these 
demographic variables with the exception of ethnicity, we noted that different indices were 
important in predicting overall satisfaction between these groups. (For convenience, the 
variables and their differing regression weights are listed in the table on the following page.) 
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Table 2 



Regression Analyses 



Overall 



- Overall Academic Satisfaction 

- Students/Social Life 

- Environment 



Adj. R 
Std Beta=.56 
.18 
.11 



.59 



By Gender 
Men: 



Women: 



Overall Academic Satisfaction 
Teaching 

Overall Academic Satisfaction 
Students/Social Life 



.67 
.22 

.55 
.27 



By College 

Commerce: 



Liberal Arts: 



Overall Academic Satisfaction 
Students/Social Life 
High School GPA 

Overall Academic Satisfaction 
Students/Social Life 



.51 
.33 
.13 

.64 
.22 



By Residence Hall Status 

Yes/In Overall Academic Satisfaction 

Students/Social Life 



.52 
.34 



No/Not in Overall Academic Satisfaction 
Environment 
Academic Support 



.53 
.24 
.12 



By Ethnicity (2-level) 

Minority Overall Academic Satisfaction 

Environment 
Cumulative GPA 



.55 
.29 
.13 



Non-Min Overall Academic Satisfaction 
Students/Social Life 



.57 
.26 



By Geographic Location 

Outside Chgo Overall Academic Satisfaction 
Students/Social Life 
Support Services 

Chgo/Sub's Overall Academic Satisfaction 
Students/Social Life 



.56 
.40 
-.21 

.63 
.21 
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1 . The important variables for men were overall academic satisfaction and 
teaching compared to overall academic satisfaction and students/social life for 
women. 

2. The important variables for students outside the Chicago/suburban area were 
overall academic satisfaction, students/social life, support services while support 
services were not important for students in the Chicago/suburban area. 

3. For Commerce students, overall academic satisfaction, students/social life and 
high school GPA were significant, but high school GPA was not significant 
for Liberal Arts Students. 

4. For students who lived in the residence halls, overall academic satisfaction and 
students/social life were significant predictors of satisfaction, but for students 
who did not live in the residence halls, overall academic satisfaction was joined 
with environment and academic support. 

5. Due to small sample sizes for different minority groups, we constructed a 
ethnicity indicator of minority vs. non-minority status. For minority students, 
academic satisfaction, environment and cumulative GPA were significantly 
important in predicting overall satisfaction but for non-minority students, 
overall academic satisfaction and students/social life were significant. 

Strengths of the Model 

One important feature of focusing on student satisfaction instead of attrition is that we 
can explore differences between student subgroups, such as satisfaction by gender or ethnicity. 
Our sample of students who responded to the student satisfaction survey and who did not 
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return to DePaul for their sophomore year was small. Because of this, our retention 
modeling using logistic regression (with the bivariate dependent variable 'retention 1 ) is limited 
to the overall freshman sample. However, using the continuous variable 'overall satisfaction 1 
as the dependent variable allows us to use the power of multiple regression to study the 
attitudes and behaviors of sub-groups within the overall population. Moreover, the focus of 
institutional efforts will be on improving services to all students, not just the 20% who leave 
after their first year. 

In addition, this analysis indicates that for different sub-groups of the student 
population, our efforts to improve student satisfaction should focus on different key variables 
(in addition to the key variable for all groups of overall academic satisfaction). For example, 
to strengthen minority student satisfaction, efforts need to be focused on environmental 
factors, such as perceptions of equal opportunities for students of all ethnic backgrounds and 
gender and developing a sense of belonging, rather than on social factors such developing 
meaningful relationships with other students and student organizations. 

To develop stronger student satisfaction for residence hall students, this analysis 
suggests that working with the Student Affairs Office to develop programs to enhance positive 
student relationships and provide meaningful social activities would be more effective, 
because of these students' close relationships with students in the residence halls, whereas 
enhancing perceptions of equal opportunities of students and a sense of belonging would be 
more effective for non-residence students. 

Limitations of Satisfaction Models 

Although these models of student satisfaction give us insight into how the components 
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of satisfaction differ among groups of students, there are limitations to this first attempt at 
modeling satisfaction. First, these are attempts at model-building. The next step in 
confirming these analyses is to test these models. Our sample sizes for the student subgroups 
were not large enough to allow us to split the sample to validate our models. In the future, 
with multiple years of data on which to draw, we will be able to conduct more powerful 
analyses. 

Second, although there were significant differences in the perceived overall satisfaction 
with DePaul between ethnic groups, because of the small samples of individual ethnic groups 
it was difficult for us to probe the group differences. As discussed above, a multi-year 
database will allow us to further explore these differences. 

Third, our satisfaction indices were intercorrelated, leading to a problem of 
multicollinearity in the multiple regression analyses. This multicollinearity in some way 
distorts the regression findings. 

Although there are limitations to these analyses, our models of student satisfaction give 
us insight into the important components of student satisfaction. As with the retention 
literature, these models suggest that academic and social perceptions are important to students 
as the weigh the impact of their educational experiences. 
Implications 

For many institutions with low attrition rates, quality enhancement programs focused 
solely at reducing student attrition may be too limited in scope. The emphasis on TQM in 
many institution has refocused energies on overall student satisfaction, recognizee; U: . 
continual improvement will reap long-term rewards in academic quality, studer. jcmi'.nent, 
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and fund-raising and institutional advancement. Institutions need to continue their services to 
cohorts of students where their may be little significant attrition. 

Moreover, given the limited number of students who leave in their first year, it was 
difficult to break the freshman cohort into smaller groups to study differences in attitudes and 
behavior. However, using satisfaction modeling, it was statistically possible to examine these 
smaller cohorts in order to develop more targeted models that can have more wide reaching 
effects than many retention programs. 

For example, De °aul found that the quality of the academic experience had wide- 
reaching effects among all cohorts of freshmen; however, other measures, such as social 
interaction and student life were effective with only some groups. Similarly, although there 
were not significant differences in the satisfaction levels between many of the cohorts of 
students, based on several demographic variables, the factors affecting student satisfaction 
were different. 

Ultimately, looking at multiple student outcomes, such as retention, graduation and 
satisfaction, can lead to more powerful institutional improvement programs and can connect 
retention efforts to a larger process of student development; more satisfied students are not 
only more likely to be retained and graduate, they are also better candidates for long-term 
institutional affiliation including support for alumni and university enhancement activities. 
Implications for Institutional Research 

1. Institutional researchers can use integrated freshm/in databases to provide 
more comprehensive outcomes assessment measures. With increasing emphasis on student 
outcomes assessment, integrated freshman databases provide a more comprehensive 
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assessment of student outcomes, including satisfaction with the academic and student service 
environment, and perceptions of integration into the community, as compared to traditional 
indicators such as freshman retention and attrition rates. Our overall model of student 
satisfaction indicated that perceptions of overall academic satisfaction was the key variable in 
predicting student satisfaction, followed by students/social life and environment. Academic 
performance variables did not enter into the equation. 

These outcome measures can be used to support outcomes assessment. Because these 
measures are more subjective, however, student perceptions may be affected by factors 
outside the control of the institution. For example, a difficult job market for law students may 
reduce satisfaction with career planning services. For this and other reasons, it is important 
for us to validate our models of satisfaction in order to be more confident that our results 
were not a function of external factors. 

2. Integrated data that provides a comprehensive picture of the freshman 
experience provides decision makers with more useful information for developing 
successful institutional strategies. For example, a study of freshman satisfaction which 
includes attitudinal data (survey data) and academic performance data provides decision 
makers with more useful information than simple satisfaction percentages for developing 
strategies to improve student satisfaction. Our analyses indicate that the focus of this 
attention should be on improving perceptions of the overall academic reputation of the 
institution by continuing to provide a strong academic program and being aggressive in 
communicating these strengths to students. 

3. Institutional researchers can develop longitudinal databases and institution- 
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specific models of the freshman experience* Multiple years: of freshman data can be linked 
in order to provide a stronger, more powerful dataset to study longitudinal trends in freshman 
attitudes and behavior. These comprehensive models of the student experiences are more 
useful to policy makers in developing strategies to improve the student experience. One 
difficulty to consider in developing longitudinal models is the changes in survey methodology 
and the survey instrument. For example, we have recently dramatically revised the student 
satisfaction survey for the ne:t: year's analysis. These improvements substantially change the 
survey, providing a more focused, less redundant study of the student experience. However, 
making longitudinal comparisons will be problematic. In this study, since we modified our 
indicator of overall student satisfaction, we were limited to one year of student satisfaction 
data. 

In addition, using theory as a guide, researchers can develop their own institution- 
specific models of student satisfaction that include the various programs and offerings related 
to the institution's uniqu' ..ssion. These institution-specific models allow policy makers to 
develop strategies that directly impact their students' unique educational experiences. 

4. This integrated analysis of the freshman experience uses a total quality 
approach to understand the freshman experience* Integrating attitudinal, behavioral and 
descriptive data sources builds on the central themes of total quality management (Sherr & 
Lozier, 1991; Coate 19° 1 ), focusing on students as important constituents and identifying 
problem areas within the university. Our satisfaction modeling recognizes the important of 
student perceptions of their educational experiences and focusses our attention on improving 
their experiences, not merely keeping students enrolled at DePaul. We can develop action 
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strategies related to these important factors to improve satisfaction can be developed and 
success can be measured via trends in multi-year student satisfaction data. 
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Endnotes 

1. Chi-Square and Mann- Whitney U tests were significant at the p<05 level. 

2. These variables entered into the step-wise logistic regression, and the Exp(B) 
were: 5.02 for in-state status; 1.0 for cumulative DPU GPA; and .47 for 
institutional commitment. 

3. Differences were significant using one-way ANOVAs p<05, with Scheffe post- 
hoc tests. 
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